Section: New Results

Gestures, Tangibles and Sound

Figure 6. Top: TouchTokens are passive tokens that guide users’ fingers to specific spatial configurations, resulting in distinguishable touch patterns. Bottom: Proof-of-concept applications: access control, tangible magic lenses, character controllers in a game, data visualization.

• We designed a new way of implementing tangible interfaces with TouchTokens [4]. The approach requires only passive tokens and a regular multi-touch surface. The tokens constrain users' grasp, and thus, the relative spatial configuration of fingers on the surface, theoretically making it possible to design algorithms that can recognize the resulting touch patterns. We performed a formative user study to collect and analyze touch patterns with tokens of varying shape and size. The analysis of this pattern collection showed that individual users have a consistent grasp for each token, but that this grasp is user-dependent and that different grasp strategies can lead to confounding patterns. We thus designed a second set of tokens featuring notches that constrain users' grasp. Our recognition algorithm can classify the resulting patterns with a high level of accuracy ($>$95%) without any training, enabling application designers to associate rich touch input vocabularies with command triggers and parameter controls.

• In collaboration with IRCAM, we introduced SoundGuides [17], a user adaptable tool for auditory feedback on movement. The system is based on a interactive machine learning approach, where both gestures and sounds are first conjointly designed and conjointly learned by the system. The system can then automatically adapt the auditory feedback to any new user, taking into account the particular way each user performs a given gesture. SoundGuides is suitable for the design of continuous auditory feedback aimed at guiding users' movements and helping them to perform a specific movement consistently over time. Applications span from movement-based interaction techniques to auditory-guided rehabilitation. We first describe our system and report a study that demonstrates a “stabilizing effect” of our adaptive auditory feedback method.